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MEASURING CAMERA ATTITUDE 



5 CLAIM OF PRIORITY 

This Application claims the benefit of U.S. Provisional Application 
No. 60/166,725, Measuring Camera Attitude, filed on November 22, 1999. That 
Provisional Application is incorporated herein by reference. 

10 BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention is directed to a system for using camera attitude 
sensors. 

15 

Description of the Related Art 

The remarkable, often astonishing, physical skills and feats of great athletes 
draw millions of people every day to follow sports. In particular, the number of 
people watching sports on television and the amount of advertising revenue 

20 received for televised sports has increased significantly. To satisfy the increased 
demand for televised sports, broadcasters have deployed a varied repertoire of 
technologies to highlight these exciting events for viewers. For example, 
broadcasters have started adding graphical enhancements to the video of the 
sporting events. Examples of graphic enhancements have included highlighting 

25 moving objects, highlighting portions of a playing field (e.g. first down line), adding 
virtual advertisements and the addition of other graphics to the video of the event. 

The systems being employed for providing graphical enhancements to video 
have generally fallen into two categories. The first category of systems uses pattern 
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recognition to recognize certain features in the video in order to accurately place 
the graphic into the video. A second category of systems uses sensors to measure 
the attitude of the camera capturing the video and then uses the measured camera 
attitude information to accurately insert the graphic into the video. It has been 

5 found that prior systems that only use pattern recognition have not been robust 
enough to account for rapid movement of the camera during the event and may be 
too slow for live events. Some systems that use pattern recognition have attempted 
to compensate for these deficiencies by using camera attitude sensors in 
combination with pattern recognition. 

10 Systems that rely on camera attitude information require precise 

measurements of the orientation of a camera at any given time. Certain situations 
beyond the broadcaster's control can interfere with and be a source of error when 
measuring camera attitude information. For example, cameras at a sporting event 
typically are located at predesignated camera locations. Sometimes the camera 

1 5 location has a floor that can sag or wobble. As a heavy camera is panned and tilted, 
the weight distribution of the camera and/or operator may cause the floor to sag or 
wobble. A camera operator moving at the camera location may also cause the floor 
to sag or wobble. Additionally, during an event, the tripod holding the camera can 
be kicked or moved. The floor of the camera location can also vibrate at either a 

20 high frequency or low frequency because of other activity in the stadium, for 
example, fans jumping, fans stomping their feet, etc. Additionally, mechanical 
compliance of the various parts of the tripod and mount can also hinder an accurate 
camera attitude reading. 

Thus, there is a need for an improved camera attitude measurement system 

25 to better measure camera attitude in light of the sources of error described above. 
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SUMMARY OF THE INVENTION 
The present invention is directed to an improved system for using attitude 
sensors with a camera. The camera can be part of a camera assembly which 
includes a movable portion and a fixed portion. One example of a camera assembly 

5 includes a tripod base, a tripod head interface mounted on the tripod base, a tripod 
head mounted on the tripod head interface and a camera mounted on the tripod 
head. In one embodiment, the system includes a first sensor coupled to the camera 
assembly and a first inclinometer coupled to the camera assembly. Instead of, or in 
addition to, the first inclinometer, the system could have a first gyroscope ("gyro") 

10 coupled to the camera assembly. The first sensor measures the position of the 
moveable portion of the camera assembly relative to the fixed portion of the camera 
assembly. In one embodiment, the first sensor is an optical encoder. In one 
alternative, the system includes two optical encoders, two inclinometers and three 
gyros. Data from the camera attitude sensors are combined to describe the 

1 5 orientation of the camera. One means for describing the orientation of the camera 
includes setting up one or more transformation matrices. Alternatively, the data 
from the various camera attitude sensors can be combined to result in a set of angles 
describing the orientation of the camera. This information can be displayed on a 
monitor, printed, stored on a computer readable storage medium or passed to a 

20 software process. 

The output of the camera attitude sensors are typically communicated to a 
camera sensor electronics package which receives the camera attitude data and 
packages the data for communication to graphics production equipment. In one 
embodiment, the data from the sensors is encoded on an audio signal and sent to the 

25 graphics production equipment (usually located remotely from the camera) via an 
audio line (or microphone line) from the camera. In one use of the present 
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invention, the graphics production equipment receives the sensor data, demodulates 
the audio and uses the camera attitude data to add a graphic to a video image from 
the camera. In one alternative, the graphic corresponds to a three dimensional 
location within a field of view of the camera. The three dimensional location 
5 corresponds to a first position in the video image, and the graphic is added to the 
video image at the first position. In one embodiment, the three dimensional location 
is converted to the first position in the video image using one or more 
transformation matrices. 

One method for practicing the present invention includes sensing data from 
10 a first sensor, sensing data from a second sensor and combining the data from the 
two sensors. In one embodiment, the second sensor can be a gyro or an 
inclinometer. The first sensor measures relative position of the movable portion of 
the camera assembly with respect to the fixed portion of the camera assembly 

Portions of the above-described process are performed using the sensors 
1 5 described above in combination with various hardware and software. The software 
for implementing the present invention can be stored on processor readable storage 
media. Examples of suitable processor storage media include RAM, ROM, hard 
disk, floppy disk, CD-ROM, flash memory, etc. In another alternative, the method 
can be performed on specialized hardware designed to specifically perform the 
20 functionality described herein. 

The hardware and software described to perform the present invention can 
be used for purposes of adding one or more graphics to live or delayed video of a 
sporting event. Alternatively, the hardware and/or software of the present invention 
can be used to determine attitude information for other purposes, for example, 
25 enhancing video of non-sporting events and for determining attitude for purposes 
other than enhancement of video. 
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These and other objects and advantages of the present invention will appear 
more clearly from the following description in which the preferred embodiment of 
the invention has been set forth in conjunction with the drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 A shows a camera mounted on a tripod. 

Figure IB shows an exploded view of the tripod head interface. 

Figure 1C is an exploded view of a portion of the tripod head interface. 

Figure ID is a bottom view of a portion of the tripod head interface, 
showing the camera lens panned at three different angles. 

Figure 2 is a block diagram of the electronics associated with the camera 
attitude sensors. 

Figure 3 is a block diagram of the interface electronics for an inclinometer 
of the present invention. 

Figure 4 is a block diagram of the audio modulation circuit used in one 
embodiment of the present invention. 

Figure 5 is a block diagram of the graphics production hardware. 

Figure 6 is a block diagram of the audio demodulator circuit of Figure 5. 

Figure 7 is a flow chart of one embodiment of a method of determining 
camera attitude information using an inclinometer with an encoder. 

Figure 8 depicts one embodiment of a method of determining camera 
attitude information using a gyro and an inclinometer. 

Figure 9 depicts one embodiment of compensating for gyro offset. 

Figure 10 depicts one embodiment of determining camera attitude 
information using a gyro and an encoder. 

Figure 1 1 is a flow chart describing one method of using camera attitude 
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DETAILED DESCRIPTION 
Figure 1A shows a camera mounted on a tripod, using the sensors of the 
present invention. Specifically, Figure 1 A shows camera 2 with lens 4 and viewing 
monitor 6. Camera 2 is mounted on tripod head 8. Tripod head 8 (also called a 
pan-tilt head) is mounted on tripod head interface 10, which is mounted on tripod 
12. Tripod head 8 allows camera 2, lens 4 and monitor 6 (collectively referred to 
as "camera") to pan and tilt. One example of a suitable tripod head is the Vector 
70 from Vinten, Inc. Although it is called a "tripod head," tripod head 8 need not 
be used with a tripod. The term tripod head is used to describe an apparatus that 
allows a camera to change its orientation (e.g. pan and tilt). Tripod head interface 
10 is mounted between tripod head 8 and tripod 12, and includes some of the 
sensors and electronics of the present invention. For example, shown in Figure 1 A 
inside tripod head interface 10 are pan encoder 1 8, inclinometer 28 and inclinometer 
30. Mounted to the outside of tripod head interface 10 is box 14, which houses 
camera sensor electronics 16. 

Tilt encoder 20 has a shaft. That shaft is coupled to top platform 86 of 
tripod head 8 (see Figure IB). Top platform 86 is mounted to camera 2. As 
camera 2 and top platform 86 tilts, the shaft of encoder 20 will rotate. In one 
embodiment, encoders 18 and 20 shown in Figure 1 are optical shaft encoders. One 
example of an appropriate encoder is the CP-850 series encoder (including CP-850- 
HCE-131072) manufactured by Computer Optical Products, Inc., 9305 Eton 
Avenue, Chatsworth, California 91311. The CP-850 series encoder is an optical 
shaft encoder. Other suitable encoders can also be used. The encoders measure 
rotational position relative to the tripod and/or tripod head interface. For example, 
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tilt encoder 20 measures movement of top platform 86 about a tilt axis in relation 
to tripod head interface 10. Pan encoder 18 measures the rotational position of 
platform 86 (as well as the camera and lens) about a pan axis in relation to tripod 
head interface 1 0. As will be discussed below, inclinometers 28 and 3 0 can be used 
5 to measure changes in the pan axis and the tilt axis. 

One example of a suitable inclinometer uses liquid between a pair of plates, 
and measures change of capacitance. Another example is an electrolyte varying the 
conductance between two conductors. In one embodiment, a suitable inclinometer 
indicates an absolute angle (relative to gravity or other acceleration). In one 

10 example, the inclinometer can indicate angles up to plus or minus one degree, plus 
or minus 1.5, degrees, or plus or minus six degrees. Other suitable ranges can also 
be used. An example of a suitable inclinometer is the Ceramic Tilt Sensor SH50054 
from Spectron, 595 Old Willets Path, Hauppaug, New York 11788, (516) 582- 
5600. Other suitable inclinometers can also be used with the present invention. 

1 5 Looking back at Figure 1 A, mounted on top of lens 4 is block 22. Mounted 

on one surface of block 22 is gyro 24. Mounted on a second surface of block 22 
is a second gyro 26. The surface that gyro 24 is mounted on is orthogonal to the 
surface that gyro 26 is mounted on. In one embodiment, an additional gyro is 
mounted on a surface orthogonal to gyro 24 and gyro 26. In another embodiment, 

20 the gyros ( two or three gyros) can be mounted on tripod 12 or on tripod head 
interface 10. For example, the gyros can be mounted to plate 80 or another portion 
of tripod interface 10. When mounted on tripod interface 10, it is advantageous to 
mount two of the gyros in a fixed parallel relation to the inclinometers, that is, one 
gyro is aligned to read data for the same axis as one inclinometer. In another 

25 embodiment, references 28 and 30 can represent both an inclinometer and a gyro. 
Mounting the gyros on tripod interface 1 0 simplifies the math used for the present 
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invention. While the encoders measure angles relative to a fixed base, the gyros 
measure absolute angular rate with respect to an internal frame. 

In one embodiment, the gyros of Figure 1 are fiber optic gyros. An example 
of a suitable fiber optic gyro is the E-Core 2000 Series Fiber Optic Gyro 
5 manufactured and sold by KVH Industries, Inc., 50 Enterprise Center, Middleton 
Road, Rhode Island 02842. Other gyros that can also be used include a ring laser, 
mechanical gyro, tuning fork, spinning disk gyro, semi conductor gyro, etc. Other 
sensors can also be used that measure change in angle or angular rate. The output 
of the E-Core 2000 Fiber Optic Gyro is an analog signal proportional to its angular 
10 rate. Integrating the output of the gyro over a period of time will produce an angle 
delta for that period of time. That is, the result of integration will indicate that the 
gyro was just rotated by a certain angle. The gyros used in Figure 1 A are single 
axis. However, multi-directional gyros or multiple gyros can also be used. In one 
embodiment, the system can use three or more gyros instead of two, in order to 
15 measure pitch, roll and yaw. The gyros can be used to measure high frequency 
vibrations and mechanical compliance between portions of the camera assembly. 
In one alternative, rather than using an angular rate gyro, the system can use an 
absolute angle gyro. 

Figure IB shows an exploded view of tripod head interface 10. In one 
20 embodiment, the components of tripod head interface 10 are surrounded by a 
housing, which is not depicted in Figure IB. The components of tripod head 
interface 10 are shown below tripod head 8 and above tripod 12. Plate 80 is 
coupled to tripod 12 using screw 32. Plate 80 is also mounted to mount 40 using 
screws 34, 36, 38 and screw 82 (screw 82 is shown in Figure 1C). Mount 40 is 
25 attached to block 50 using screws 44, 46 and 48. Mounted inside block 50 is pan 
encoder 18. Block 50 includes a triangular portion which protrudes out from the 
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bottom of block 50 (see Figure 1C) causing a separation between block 50 and 
mount 40. Residing in the space between block 50 and mount 40 is arm 56 which 
is connected to member 54 using screw 60. Located between member 54 and arm 
56 is a self aligning spacer 58. 

As will be shown in Figures 1C and ID, arm 56 is coupled to the shaft of 
pan encoder 18. Member 54 is coupled to tripod head 8. As tripod head 8 rotates 
(pans), arm 56 also rotates, which causes the shaft of pan encoder 18 to rotate, 
thereby, allowing pan encoder 18 to measure the rotation or pan of the camera. 

Mounted on one surface of block 50 is inclinometer 30. Mounted on a 
second surface of block 50 is a second inclinometer 28. The surface that 
inclinometer 28 is mounted on is orthogonal to the surface that inclinometer 30 is 
mounted on. Mounted in front of inclinometer 30 is a PC board 84. Inclinometer 
28 and inclinometer 30 are both connected to PC board 84. In one embodiment, 
PC board 84 includes electronics that are in communication with camera sensor 
electronics 16. Block 50 includes four holes 72, 74, 76 and 96 (hole 96 is shown 
inFigure 1C) that house screws for mounting block 50 to plate 100 (see Figure 1C). 

As described above, one embodiment that simplifies the math includes 
mounting the gyros on plate 80. In one embodiment, plate 80 depicted inFigure IB 
would be re-designed so that it was a unitary structure with two portions. The first 
portion would be circular with mounting holes as depicted in Figure IB. The 
second portion would include a beam protruding from the circular portion. In one 
alternative, the beam would extend six inches. Three gyros are mounted on the 
beam such that the three gyros are mounted orthogonal to each other. 

Figures 1 C shows additional details of tripod head interface 10. Specifically, 
Figure 1C shows arm 56 engaging shaft 98 of encoder 18. At the end of arm 56 is 
end portion 90 coupled to arm 56 via screws 92 and 94. Member 54 is mounted on 
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plate 1 02 via two screws that are not shown in the drawings. Plate 1 02 is mounted 
to tripod head 8 so that when tripod head 8 rotates (pans) then member 54 and arm 
56 are also rotated about the pan axis, causing shaft 98 to rotate and allowing 
encoder 18 to measure that rotation. The screws residing in holes 72, 74, 76 and 
5 96 mount block 50 to plate 100. Plate 100 is coupled to plate 102 using bearings 
so that plate 102 will rotate with respect to plate 100. In the embodiment depicted 
in Figures 1A-1D, plate 80, mount 40, block 50 and plate 100 are stationary with 
respect to tripod 12. As tripod head 8 is rotated about the pan axis, the camera and 
plate 102 also rotate. 

10 Figure 1 C also shows more detail of the coupling of mount 40 to block 50. 

Specifically, triangular portion 88 and block 50 are shown in more detail. Because 
triangular portion 88 is raised from the remaining portion of the bottom surface of 
block 50, there is sufficient space between block 50 and mount 40 to allow for the 
presence of arm 56. Additionally, block 50 is contained within a housing. 

15 Figure ID shows lens 4 in three different positions (4, 4' and 4"). As can 

be seen, as lens 4 is moved to position 4', arm 56 is rotated to position 56'. As lens 
4 is rotated to position 4", arm 56 is rotated to position 56". As arm 56 is rotated 
to position 56' and 56", shaft 98 is similarly rotated. 

Figure 2 shows a block diagram of the electronics for using the camera 

20 attitude sensors. Figure 2 shows pan encoder 18, tilt encoder 20, gyro 24, gyro 26, 
inclinometer 28 and inclinometer 30. The output of pan encoder 1 8 and tilt encoder 
20 are sent to FPGA 212. Pan encoder 1 8 and tilt encoder 20, in one embodiment, 
are optical encoders that output a signal, measured as a number of counts (or 
pulses), indicating the rotation of shaft. The output signal is a quadrature signal 

25 indicating rate and direction. FPGA 212 decodes the signal from the shaft encoders 
to output a count. FPGA 212 also controls analog to digital converter 214 and 
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provides interface logic for processor 216. In regard to the analog to digital 
converter 2 1 4, FPGA 2 1 2 provides interface logic and a buffer, including a register 
to store a value for each sensor connected to analog to digital converter 214. 

Gyro 24 is connected to interface board 220, which is connected to analog 
5 to digital converter 214. Interface board 220 comprises electronics for receiving 
a signal from gyro 24 and presenting the information to analog to digital converter 
214. The electronics of board 220 includes a differential amplifier and other 
electronics which can reject common mode noise and amplify the signal from the 
gyro. The output of gyro 26 is connected to interface board 222. Interface board 
10 222 operates in the same manner as interface board 220 and is also connected to 
analog to digital converter 214. 

Signal 224 represents the electrical output of the zoom lens potentiometer 
of the camera and is connected to analog to digital converter 214. Signal 226 
represents the electrical output of the 2X extender of the camera and is connected 
15 to analog to digital converter 214. Signal 228 represents the connection to the lens 
of the camera, provides the value of the focus of the camera and is connected to 
analog to digital converter 214. 

The output of inclinometer 28 is connected to interface board 230. The 
output of inclinometer 30 is connected to interface board 232. The outputs of 
20 interface board 230 and interface board 232 are both connected to analog to digital 
converter 214. Analog to digital converter 214 converts the input analog signals 
to digital signals, and sends the output digital signals to FPGA 212. FPGA 212 
includes a register for each of the sensors. In one embodiment, the electronics of 
interface boards 230 and 232 are included on PC board 84. In one alternative, PC 
25 board 84 can include electronics and LEDs to indicate when tripod head 8 is level. 

Processor 216 is in communication with data memory 236 for storing data 
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and program memory 238 for storing program code. In one alternative, memory 
238 is a flash memory and memory 236 is a static RAM. In one embodiment, 
processor 216 is an 8032 processor from Intel. Processor 216 also receives an 
output signal from sync decoder 240. Sync decoder 240 receives a video signal 250 

5 from camera 2. Sync decoder 240 generates a sync signal so that the data from the 
sensors can be synchronized to the video. In one embodiment, the video is 
transmitted at 30 frames per second. Other video rates can also be used. Processor 
216 assembles data from each of the sensors into a packet and sends the data to 
modulator 244 . Processor 2 1 6 assembles the data using the sync signal so that data 

10 is collected and sent in synchronization with the video from the camera. For 
example, data can be sent for every field, every video frame, every other video 
frame, every third video frame, etc. In one embodiment, the packet of data sent 
from processor 216 does not include time code or any type of synchronization 
signal. 

1 5 Modulator 244 receives the packet of data from processor 2 1 6 and encodes 

data for transmission on an audio frequency signal. The output of modulator 244 
is sent to audio driver 246 and coax driver 248. Most broadcast cameras have a 
microphone input channel. The output of audio driver 246 is sent to the 
microphone input channel for camera 2. The camera then combines the audio input 

20 channel with the video and sends a combined signal to the production equipment. 
If the audio signal is needed on a coax cable, then that signal is received from coax 
driver 248. In one embodiment, there can also be an RS232 or RS422 output 
directly from processor 216. 

Figure 3 is a block diagram of interface boards 230 and 232. Note, in one 

25 embodiment, interface board 230 and 232 can be combined as one board with two 
inputs and two outputs. Figure 3 shows sensor 300, which can be inclinometer 28 
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or inclinometer 30. One input to sensor 300 is oscillator 302, which provides a 2.5 
KHz square wave. Sensor 300 has two outputs. One output is connected to one 
end of resistor 304 and the positive terminal of differential amplifier 308. The other 
output is connected to one end of resistor 306 and the negative terminal of 
5 differential amplifier 308. The other ends of resistor 304 and 306 are connected 
together and to ground. The output of differential amplifier 308 is sent to multiplier 
310. The output of oscillator 302 is also sent to multiplier 310. In one 
embodiment, multiplier 3 10 operates to convert the output of amplifier 308 to a DC 
signal. The output of multiplier 3 10 is sent to low pass filter 3 12 which removes 

10 noise, specifically the 2.5 KHz oscillator signal. In one embodiment, low pass filter 
312 has a 40 Hz cut off frequency. The output of low pass filter 3 12 is sent to 
amplifier 314. The output of amplifier 3 14 is a linear voltage output indicating 
inclination angle. In one embodiment, the dynamic range of the circuit of Figure 3 
is plus or minus 1.5 degrees. 

15 Figure 4 is a block diagram of modulator 244 of Figure 2. The input of 

modulator 244 is first sent to FEC and Interleave unit 340, which receives the data, 
adds forward error correction and error detection information (cyclic redundancy 
check - CRC) and interleaves the result for burst-error protection. The output of 
FEC and Interleave unit 340 is sent to data pump 342, which converts the data into 

20 a four level analog signal. The output of data pump 342 is sent to low pass filter 
344. The output of data pump 342 is said to be in the form of level coded symbols. 
The output of low pass filter 344 is sent to a voltage controlled oscillator 348 which 
is used to modulate an audio signal with a 10 KHz center frequency (also called FM 
for "frequency modulation"). The output of voltage controlled oscillator 3 48 is sent 

25 to amplifier 350. The output of amplifier 350 is sent to transformer 352. The 
output of transformer 352 is sent to the microphone input channel of a video camera 
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via audio driver 246. Figure 4 shows box 346 drawn around FEC and Interleave 
unit 340, data pump 342 and low pass filter 344. In one embodiment, the functions 
inside box 346 are performed by a MX919 4-Level FSK Modem Data Pump from 
MX-COM, Inc., 4800 Bethania Station Road, Winston- Salem, North Carolina 
5 27105, (800) 638-5577, www.mxcom.com. 

Figure 5 is a block diagram of the graphics production equipment. In one 
embodiment, the equipment depicted in Figure 5 is used in a truck at an event. 
Alternatively, the equipment can be in a studio or another suitable location. Audio 
demodulator 400 receives audio signal(s) from one or more cameras. In one 
10 embodiment, it receives audio signals from three cameras. In alternative 
embodiments, audio demodulator 400 can receive audio signals from less than three 
cameras (even one camera) or more than three cameras. Each of the audio signals 
includes data from the sensors associated with the camera providing the audio 
signal. Audio demodulator 400 extracts the data from the audio signal and sends 
1 5 the data for each of the associated cameras to concentrator 402 . Concentrator 402 
receives the camera attitude data for sensors from one or more cameras and 
packages the data for transmission to computer 406. In one embodiment, 
concentrator 402 is a computer. Concentrator 402 also receives a time code from 
time code generator 404. The time code is packaged with the data from the various 
20 sensors so that the data can be synchronized with the appropriate frame or field of 
video. In one embodiment, time code 404 is a VITC (Vertical Interval time Code) 
inserter. Time code generator 404 receives the tallied video signal and provides a 
time code, based on the received video, to concentrator 402. After time code 
generator 404, the video signal is sent to computer 406 and delay 410. In one 
25 embodiment, a time code and a unique camera identifier are added to the vertical 
blanking interval of the video from the instrumented cameras. In another 
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embodiment, a time code is added to the sensor data at the camera location, prior 
to transmission to audio demodulator 400. Concentrator 402 also receives key data 
from another processor. The key data is used to indicate which colors in the video 
can be enhanced or blended with the new graphic. This allows the system to 
5 account for occlusions. More information about using key data can be found in 
U.S. Patent Application Serial No. 09/160,534, System For Enhancing a Video 
Presentation of a Live Event, Gloudemans, et al. and U. S. Patent Application Serial 
No. 09/425,992, filed on October 21, 1999, Attorney Docket No. 
SPTV-01038US0 BBM), Telestrator System, Meier, et al. both of which are 
10 incorporated herein by reference. 

The combined data from the sensors is sent to computer 406. Computer 
406, computer 408, delay 410 and keyer 412 are used to enhance live video from 
a chosen camera. The present invention works with various systems for enhancing 
the video. For example, suitable systems are described in the following 
15 patents/applications: U.S. Patent No. 5,912,700, A System for Enhancing the 
Television Presentation of an Object at a Sporting Event, U.S. Patent No. 
5,9 17,5 53 , Method And Apparatus For Enhancing The Broadcast of a Live Event, 
U.S. Patent Application Serial No. 09/041,238, System For Determining The 
Position Of An Object, filed January 6, 1998, U.S. Patent Application Serial No. 
20 09/1 60,534, A System For Enhancing a Video Presentation of a Live Event, filed 
September 24, 1998, all of which are incorporated herein by reference. 

Computer 406 receives the sensor data, key data and time codes from 
concentrator 402. Computer 406 also receives the video signal, including VITC. 
In one embodiment, computer 406 is used to choose a location on the playing field 
25 of a sporting event. The location can be chosen using any suitable means including 
a pointing device, a keyboard, or a software process. The three dimensional 
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coordinates associated with the chosen position are determined using any number 
of means in the art including using a model, prestoring locations, manually entering 
the locations, sensors (infra-red, radar, . . .) etc. Using the data from the camera 
attitude sensors, computer 406 converts the three dimensional locations) of the 

5 chosen location to two dimensional position(s) in the frame or field of video from 
the chosen camera. That two dimensional position(s) are sent to computer 408 to 
draw (or set up) a field or frame of video with the graphic. Computer 408 then 
sends instructions to a keyer to combine the graphic with the video from the 
camera. The video from the chosen camera is sent first to delay 410 in order to 
10 delay the video a number of frames to allow for the processing of the camera 
attitude information and the other methods of computers 406 and 408. After being 
delayed, the video is sent from delay 410 to keyer 412 for combining with the 
graphic(s) generated by computer 408. The output of keyer 412 can be sent for 
broadcast or recorded for future use. In the embodiment described above, the 

15 components operate in real time and enhance live video. In another embodiment, 
the camera attitude information can be used to enhance pre-stored video. An 
example of a graphic can be a line added to a video of a football game; a virtual 
advertisement; a cloud to show the location of a moving object; or any other 
suitable graphic. 

20 In one embodiment, a third computer can be added to computers 406 and 

408. The third computer can be used to provide a user interface which, among 
other things, allows an operator to choose which colors can be replaced with a 
graphic. This third computer would supply the key data to concentrator 402. In 
this embodiment, computer 406 determines where within a given field or frame a 

25 graphic should be inserted and computer 408 draws the graphic and synchronizes 
the graphic with the appropriate field or frame. 
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Figure 6 is a block diagram of audio demodulator 400. Data is first sent to 
phase-locked-loop circuit 480. The output of phase-lock-loop circuit 480 is sent 
to filters 482. The output of filters 482 is communicated to low pass filter 484. 
The output of low pass filter 484 is sent to signal decoder 486, which reverts the 

5 four level analog signal back to a digital signal. The output of signal decoder 486 
is sent to FEC Interleave module 488 which completes the forward error correction 
and accounts for the interleaving. The output of FEC and Interleave module 488 
is sent to concentrator 402. Box 490 is drawn around low pass filter 484, signal 
decoder 486 and FEC and Interleave module 488 to indicate that all three functions 

1 0 can be combined into one chip. For example, all three functions can be performed 
by using the MX919 chip from MX-COM, described above. The circuit of Figure 
6 is used to operate on data for one camera. If data from multiple cameras is 
considered, then audio demodulator 400 will include multiple circuits like that of 
Figure 6. 

15 Figure 7 is a flow chart describing one method for using the inclinometers 

with an optical encoder. In step 560, voltages are read from each of the 
inclinometer sensors (e.g. 28 and 30). In one embodiment, step 560 includes a 
signal being processed by interface boards 230 and 232 to analog to digital 
converter 214, and through FPGA 212 to processor 216. Processor 216 will read 

20 the appropriate values at the appropriate time according to the sync signal sent from 
sync decoder 240. In step 562, the data is passed through a low pass filter. In one 
embodiment, the low pass filter can be performed using hardware. In other 
embodiments, computer 406 can implement a low pass filter using software. The 
low pass filter could include averaging data over a period of time. For example, 

25 samples from 30 frames of data can be averaged. In one alternative, the data can 
be averaged by looking at 1 5 samples ahead in time and 1 5 samples behind in time. 
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The reason why there are samples ahead in time is because the video is delayed by 
frame delay 410. Other time frames other than samples of 30 frames or 30 fields 
can also be used. 

In step 564, the values read from the inclinometers are converted to a 

5 camera attitude parameter for use by the production equipment. For example, the 
voltages can be converted to angles. One means for converting voltage to an angle 
is to use a look up table. In another embodiment, a scaling factor can be applied to 
the voltage to convert the voltage to an angle. The angle can be expressed in 
degrees, radians or another unit of measure. Rather than converting the voltage to 

10 an angle, the voltage can be converted to a different form of an attitude parameter 
(e.g. such as a variable in a matrix). 

In step 568, the system reads the encoder. Note that step 568 may be 
performed at the same time as step 560. Step 568 includes FPGA 212 receiving 
data from one of the encoders and providing that data to processor 216. Processor 

15 216 will read the data in accordance with the sync signal from sync decoder 240. 
In step 570, the data from the encoder is converted to a camera attitude parameter 
for use by the production equipment. For example, the voltage can be converted 
to an angle. In step 572, the parameter from the encoder is combined with the 
parameter(s) from the inclinometer(s). Remember that the encoder measures the 

20 amount of rotation of the camera with respect to the base. In one embodiment, the 
inclinometers measure attitude of the base. Thus, the actual orientation is 
determined by using information from both encoders and both inclinometers. The 
inclinometers can be thought of as measuring roll and pitch of the tripod, while the 
pan encoder measures pan angle of the camera with respect to the base of the tripod 

25 or in relation to the tripod head interface. The pan axis is moved from a completely 
vertical axis to a different orientation based on roll and/or pitch. A similar analysis 
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applies for tilt. One method for performing step 572 is to create one or more 
transformation matrices (to be used in step 712 of Fig. 1 1). Another embodiment 
uses different means known in the art for summing angles from different sensors. 
Figure 8 explains one method for combining a gyro with an inclinometer. 
5 The embodiment of Figure 8 assumes that the gyros are mounted on the tripod head 
interface 1 0 such that for each inclinometer there is a gyro mounted to measure data 
for the same axis. One gyro and one inclinometer are mounted on the pitch axis. 
Another gyro and inclinometer are mounted perpendicular to the pitch axis devices 
in order to measure the roll axis. Each gyro and inclinometer pair are independently 
10 used to measure pitch and roll. An additional gyro or gyro/inclinometer pair can 
also be used to measure yaw. The system of Figure 8 is used to measure pitch. An 
identical system is used to measure roll. 

Gyro 6 1 0 is a fiber optic gyro which measures angular rate. It does not have 
any reference to absolute angle. It can accurately measure the relative angular 
15 change. The output of gyro 610 is integrated using a digital or analog integrator 
614. The output of integrator 614 will represent an angle. The output of integrator 
614 is scaled (block 624). We can describe the measured output of the integrated 
gyro signal as Aq = A + e G where A^ = integrated angle, A = actual angle gyro 6 1 0 
was rotated and e G = error induced by gyro 610. The error e G is largely due to the 
20 offset drift of the gyro and only has low frequency components. 

Inclinometer 604 can accurately measure the true pitch or roll of the camera 
mount but is subject to acceleration errors. Gravity and acceleration of a reference 
frame are indistinguishable, so it is difficult to tell the difference between gravity 
indicating which way is "down" and acceleration imposed on the sensor. Imagine 
25 that the camera mount is not disturbed and is on a very stable platform. The 
inclinometer will accurately read the angle of the camera mount because the only 
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force acting on the inclinometer is gravity. If the camera mount is tilted, it will 
accurately measure the new pitch or roll. During the transient when rotating the 
camera mount, acceleration will be induced on the sensor unless the axis of rotation 
is precisely through the inclinometer sensor and the pan axis is nearly frictionless. 
5 The axis of rotation will typically not be through the inclinometer, so changes in 
pitch or roll will induce a transient error in the inclinometer reading. In addition to 
this error, the device being used has a slow response to transients. If the device is 
rotated rapidly about the axis so as not to induce acceleration errors, the response 
time is about one second. If we think about the inclinometer signal output errors 
10 in the frequency domain, we can say that the low frequency errors are very small 
because the average acceleration will be near zero as long as we are not translating 
the inclinometer to a new position. Most of the inclinometer errors will be high 
frequency due to transient accelerations. Let A, = A + e t where A x = measured 
angle from inclinometer, A = actual angle inclinometer 604 was rotated and ^ = the 
1 5 inclinometer measurement error due to acceleration and sensor response. The error 
ej will have very little low frequency components. 

Summer 616 will subtract (output of scale block 624) from A, (output 
of inclinometer 604) yielding e, - e G . This signal is passed through low pass filter 
(LPF) 618. The cutoff frequency ofLPF 618 is chosen to pass the gyro error signal 
20 e g but reject inclinometer error signal e r A typical cutoff frequency is 0.2Hz. The 
output of LPF 618 will be -e G . Summer 620 will add signal Aq from scale block 
624 to -e G from LPF 618. The result is signal A, the desired actual angular rotation. 

An enhancement to the method is to adaptively set the cutoff frequency of 
LPF 618. If the system is not experiencing any acceleration, it is advantageous to 
25 raise the cutoff frequency to reduce errors due to drift in the gyro. If the system is 
experiencing accelerations, it is advantageous to lower the cutoff frequency to 
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reduce the errors due to the inclinometer. Acceleration errors will be seen as a high 
frequency signal at the output of summer 616. The output of summer 616 is sent 
to high pass filer (HPF) 626. The output of HPF 626 is then sent to a fast attack, 
slow decay detector 628. The output of detector 628 is used to set the cutoff 

5 frequency of LPF 618. 

Figure 9 shows one embodiment for automatically compensating for gyro 
offset correction. If the gyro is not rotating, it will have some non-zero signal. If 
this non-zero signal is integrated, the resulting signal will be a ramp signal. In the 
embodiment where the gyro is mounted to a fixed portion of the camera assembly, 

10 over the long term the gyro is not rotating through a net angle. The average angle 
change is zero. Figure 9 shows the output of gyro 6 1 0 being sent to subtractor 63 0. 
The output of subtractor 630 is sent to integrator 614 and amplifier 632. The 
output of amplifier 632 is sent to integrator 636 and the output of integrator 636 
is subtracted from the output of gyro 610 by subtractor 630. Subtractor 630, 

15 amplifier 632 and integrator 636 comprise a feedback system that generates an 
offset signal that cancels the gyro offset. After a period of time determined by the 
gain of amplifier 632, the signal at the output of integrator 636 will be equal to the 
offset of gyro 610. 

In another enhancement, a gyro may be placed on the stationary portion of 
20 the camera assembly (e.g. tripod, tripod head interface) so it is sensitive to the pan 
axis. If the camera is panned very quickly, the camera mount may twist. The pan 
encoder 618 will not measure the amount of twist. A gyro will measure this twist 
and a correction can be applied to the resulting pan axis. 

Figure 10 shows how the gyro may be high pass filtered and combined with 
25 the pan axis encoder. The output of gyro 610 is sent to integrator 640. The output 
of integrator 640 is sent to high pass filter 642. The output of high pass filter 642 
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is added (summer 644) to the output of pan encoder 618. 

Figure 11 is a flow chart which describes how the camera attitude 
information is used by the system for enhancing video. In step 700, the system or 
an operator determines the location to be enhanced. That could be a location on a 

5 playing field, in a stadium or another location. For example, if the system is to 
enhance a broadcast of a football game by depicting a first down line (e.g. the line 
a player must cross to achieve a first down) or to add an offside line to a soccer 
telecast, the system or person can indicate the location on the field where the line 
should be drawn using any conventional input device (e.g. keyboard, pointing 

10 device, etc.). That location can be associated with one or more three dimensional 
coordinates. In one embodiment, the location is the current location of a moving 
object. In step 702, attitude data is read from the various attitude sensors (e.g. 
inclinometer, gyro, encoder, etc.). In many broadcasts of sporting events, multiple 
cameras are used. Thus, attitude sensors may be used with one or more cameras. 

15 Step 702 includes reading the attitude data for all the cameras. In some 
embodiments, step 702 can include reading data for only a subset of the cameras. 
In step 704, the attitude data for the cameras is added to the respective audio signal 
for the corresponding camera. In step 706, the system determines the tally. The 
tally indicates which camera is being used for broadcast. That is, there may be 

20 twelve cameras at a sporting event; however, usually only one camera is chosen for 
broadcast at any one given time. The system can determine the tally by having an 
automatic tally detector, looking for a unique camera identifier in the vertical 
blanking interval or reading tally information from other production equipment. 
One could even enter the tally manually by watching or listening to the director, etc. 

25 By knowing which camera is tallied, the system can determine which set of camera 
attitude data to use. That is, only the camera attitude data associated with the 
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camera tallied is used to enhance the video. 

In step 708, the attitude information is extracted from the audio signals. In 
one embodiment, attitude information for all of the cameras are extracted from all 
the respective audio signals by audio demodulator(s) 400 and sent to concentrator 
5 402 which then sends the combined signal to computer 406. In another 
embodiment, only the data from the tallied camera is extracted. In step 710, the 
system determines the attitude of the tallied camera. Determining the attitude can 
include the teachings of Figures 7-10. In one embodiment, the data from each of 
the sensors can be converted to appropriate parameters for insertion into 
10 transformation matrices. In step 712, the location(s) determined in step 700 are 
transformed to a two dimensional position in the frame or field of video from the 
tallied camera. Step 712 includes multiplying the determined location(s) by one or 
more transformation matrices, where the parameters of the transformation matrices 
are determined from the camera attitude data. In step 714, the video is enhanced 
15 by placing a graphic at the two dimensional location(s) determined in step 712. 

The following discussion provides more detail in regard to steps 710 and 
712 for an exemplar system using inclinometers, and/or gyros, with encoders. To 
convert a three-dimensional location to a two dimensional position, a four-by-four 
transformation matrix [M^J maps a four dimensional row vector representing the 
20 three dimensional location in world coordinates into a two dimensional position in 
camera coordinates. 

There are four coordinate systems to consider: world coordinates, three 
dimensional camera coordinates, two dimensional camera (screen) coordinates and 
roll/pitch coordinates. The roll/pitch coordinates define the coordinate system with 
25 the inclinometers at or near the origin. The pan axis may or may not be vertical. 
The tilt axis may or may not be horizontal. Roll and pitch will describe the direction 
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vector of the pan axis and the tilt axis. 

The world coordinates are the coordinates of the playing field or event. In 
the system for enhancing a football game, the world coordinates can be a system 
with the origin in the corner of the football field. The positive x-axis in world 
coordinates is defined by the sideline closest to the cameras, the positive y-axis in 
world coordinates is defined by the goal line to the left of the direction of the 
cameras and the positive z-axis points upward, perpendicular to the x-axis and y- 
axis. If the positive y-axis in world coordinates is rotated counter-clockwise <D 
degrees around the positive z-axis so that the new positive y-axis is pointing in the 
same direction as the camera when the tilt of the camera is level (perpendicular to 
the direction of gravity) and the camera's pan encoder measures zero, then the pan 
offset for that particular camera is <D. For the analysis below, assume that the roll 
is represented by the variable p and the pitch is represented by the variable y. The 
values p & \|/ are the output A of summer 620 of Fig. 8. Roll and pitch are 
measured by the inclinometers alone, gyros alone, or a combination of inclinometers 
and gyros. Assume that because the roll and pitch changes the orientation of the 
pan axis (and perhaps the tilt axis), the new unit direction vector of the pan axis is 
given by the vector (v l5 v 2 , v 3 ). Then, the following equations define variables used 
in the matrices. 



( v i> v 2> v 3) ' I 1 * 0 * 0 ) = Vj= cos(90°-p) = sin(p) 



(vy,v 2 ,v 3 ) ■ (0,1,0) =v 2 = -cos (90° -\|/)= -sin(y) 
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(0,0,1) • (v p v 2 ,v 3 )=v 3 
T= arccosjv^j 

5 Additionally, assume that the origin of the roll/pitch axis (location of a fixed 

point near inclinometers and/or gyros) in world coordinates is (rpx, rpy, rpz). Using 
these variables, the matrix [M^J is created by multiplying a first matrix [M^] by 
a second matrix [M^, c ] . The matrix [M^J represents a transformation from world 
coordinates to roll/pitch coordinates. The matrix [M^ J represents a transformation 
10 from roll/pitch coordinates to two dimensional camera coordinates. The matrices 
are defined as follows: 

KJ - K.-J x K.J ■ 

The matrix [M^] is defined by 

15 

where 

1 0 0 0 
0 10 0 
0 0 10 
-rpx -rpy -rpz 1 

20 
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cos(O) -sin(€») 0 0 

sin(O) cos(<E>) 0 0 

0 0 10 

0 0 0 1 



-Vj 2 +cos(7) + 1 
(l+cos(7)) 



(1 +cos(2)) 



(1 +cos(7)) 
-v 2 2 +cos(7) + l 
(1 +cos(J)) 



cos(7) 
0 



The matrix [M^ J is defined by 



1 0 

0 1 
0 0 



0 0 

0 0 

1 0 



rpx rpy rpz 1 
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cos(<I>) sin(O) 0 0 
-sin(<I>) cos(O) 0 0 



0 



0 



1 0 



0 



0 



0 1 



The matrix [R^] is the four-by-four matrix corresponding to a counter- 
clockwise rotation by O degrees around the positive z-axis. The matrix [T^] 
denotes a four by four matrix corresponding to a translation by (rpx, rpy, rpz). The 
5 matrix [R*] is the inverse of the matrix which provides a transformation from 
roll/pitch coordinates to world coordinates. The matrix [K] represents a 
transformation matrix for transforming a three dimensional location in world 
coordinates to a two dimensional position in camera coordinates for a system that 
uses encoders but does not use inclinometers or gyros, and is similar to the 
10 transformation matrices described in U.S. Patent Application No. 09/160,534 and 
U.S. Patent No. 5,912,700, both of which are incorporated by reference. The 
matrix [K] is defined as: 



[K] = [T][A][D][B][C][G] 



15 



where the matrix [G] is defined as 



[G] = 



' jh 0 0 0" 

0 Jv 0 0 

0 0 -(f + n )/(f-n) -1 

0 0 -2fn/(f-n) 0_ 



Attorney Docket No.: SPTV-01035US0 BBM 
/bbm/sptv/1035/1035.001 



-28- 

The matrix [G] models the effective focal length of the lens as a function of 
zoom, focus, and 2X Extender settings. The variables n and f are the distances to 
the mathematical near and far clipping planes, which are only important in assigning 
a useful range for z-buffered graphics drawings; therefore, nominal values are used 
5 of n=l yard and f = 100 yards. The variable fh is the effective horizontal focal 
length of the lens. The variable fv is the effective vertical focal length of the lens. 
The aspect ratio, which is constant, is fv/fh. A software routine is used to convert 
the appropriate zoom factor and aspect ratio to fh and fv. 
The matrix [A] is defined as: 



[A] = 



cos(pari) 
sin(pari) 

0 

0 



- sm(pari) 0 0 

coupon) 0 0 

0 1 0 

0 0 1 



The matrix [B] is defined as: 



[B] 



cosipvisf) 0 sin(twist) 0 

0 10 0 

sm(twist) 0 cos(twist) 0 

0 0 0 1 



20 
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The matrix [C] is defined as: 



"l 


0 


0 


0" 


0 


0 


-1 


0 


0 


1 


0 


0 


0 


0 


0 


1 



The matrix [D] is defined as: 



[D] = 



1 0 

0 cos(tilt) 

0 sm(tilt) 
0 0 



0 0 

- sm(tilt) 0 

cos(tilt) 0 

0 1 



The matrix [T] is defined as 



m = 



1 


0 


0 


0 


0 




0 


0 


0 


0 


1 


0 


-Ix 


-fy 


-Iz 


1 



The parameters in the above described matrices are discussed below. 
10 The pan parameter is defined as (pan = pan/reg - ptzfdit.pan), where 

ptzfdit.pan is measured with the pan optical encoder during the event. The variable 
pan/reg is determined using the pan optical encoder prior to the event. First, the 
camera's optical center is pointed to a known fiducial. A known fiducial is a 
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marking or location whose coordinates are known by accurately measuring the 
coordinates in relation to the origin. The coordinates of a fiducial can be measured 
using a laser plane, tape measure, and/or other suitable methods. The pan encoder 
reading in degrees (0) is noted. The x,y coordinates of the fiducial (xl,yl) are 
5 noted. The x, y coordinates of the camera are noted (x2,y2). An angle a is 
determined as: 

OL=tarr 1 ((yl-y2)/(xl-x2)). 

10 The pan registration variable is computed as: 

pan/reg = 180°-9-a. 

The tilt parameter is defined as (tilt = ptzfdit.tilt - level tilt,), where 
15 ptzfdit.tilt is measured with the tilt optical encoder during the event. The variable 
level_tilt represents the output of the tilt optical encoder at level tilt. Level tilt is 
the tilt of the camera when the optical axis is perpendicular to the force of gravity. 
Level tilt is found by setting a laser plane next to the camera at the level of the 
camera's lens. A stick or other object that can be used to view the marking from 
20 the laser plane should be placed across the stadium at a height to receive the beam. 
By pointing the optical center of the camera on the point illuminated on the stick by 
the laser plane across the stadium, the camera is brought to level tilt. The level_tilt 
parameter is the encoder reading, in degrees (or radians) at level tilt. 

The twist parameter is determined by pointing the camera at the field (or 
25 other portion of an environment) and the output of the camera is sent to a 
computer. The image of the camera is superimposed over a transformed image of 
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a model of the environment. A slider on a graphical user interface (GUI) is used to 
alter the twist of the camera image so that it completely aligns with the image of the 
model. The degree of alignment correction is recorded as the twist parameter. 
Note that the transformation of the image of the model is performed with the best 
5 parameters known at the time. 

The nodal_dist variable (used below) is the distance from the pan axis to 
the nodal point of the camera model. The distance is positive in the direction of the 
camera along the optical axis through the front piece of glass on the lens of the 
camera. The nodal point is the position of the camera's virtual point of view 

1 0 measured as a distance forward of the pan axis when the camera is in the horizontal 
position. The variable, nodaldist, changes for different zoom percentages and 
extender settings of the camera. The manufacturer of the lens can provide values 
that determine nodal_dist at different zoom percentages and extender settings. In 
one example, the manufacturer of the lens can provide a table of the distance of the 

1 5 nodal point from the front piece of glass on the lens for each extender setting and 
a range of zoom percentages. For example, if the distance of the nodal point from 
the front piece of glass on the lens is dp yards, and the length of the lens is lens len 
yards, then nodal_dist = lens_len - dp, where nodal dist is measured in yards. If 
data from the manufacturer of the lens is not available, the information can be 

20 measured on an optical bench and a lookup table built as a function of zoom 
position, focus, and 2X Extender setting. The information of the lookup table is 
measured by placing two targets in the view of the camera, off- center, one farther 
away than the other, so they appear in line through the viewfmder. Where a line 
extended through those targets intersects the optical axis of the camera is the 

25 position of the nodal point or virtual point of view. 

The coordinates (cx,cy,cz) are the world coordinates of the location of the 
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camera, which is defined as the intersection of the pan axis and the optical axis 
when the tilt is level and the pan measures zero on the pan encoder. The 
coordinates (lx,ly,lz) are the world coordinates of the nodal point of the camera 
model for a given tilt, pan, zoom, and extender setting of the camera. The 
coordinates (lx,ly,lz) are defined by: (lx,ly,lz) = (cx,cy,cz) + (nx,ny,nz), where 
(nx,ny,nz,l) = (0,nodal_dist,0,l)[Rx,tilt][Ry,pan]. The matrix [Rx,tilt] is defined 
as: 



[Rx,tilt]= 



10 0 0 

0 cos(tilt) sm(ti!t) 0 

0 -sm(tilt) cos(tilt) 0 

0 0 0 1 



and the matrix [Ry,pan] is defined as 



[Ry,pan]= 



cos(pari) 0 -sm(pari) 0 

0 10 0 

sm(pan) 0 cos(pan) 0 

0 0 0 1 



After using the transformation matrices, the system takes into account lens 
distortion. That is, each two-dimensional pixel position is evaluated in order to 
determine if the two-dimensional position should change due to lens distortion. For 
a given two-dimensional pixel position, the magnitude of a radius from the optical 
center to the two-dimensional pixel position is determined. Lens distortion is 
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accounted for by moving the pixel's position along that radius by an amount aR: 
aR = K{R) 2 

where 

R = pixel distance from optical center to two-dimensional position 
K = distortion factor. 

5 At a fixed focus, the distortion factor is measured at a number of zoom 

values using a GUI slider to align the model to the video. These values are used to 
generate a distortion curve. During operation, the distortion factor at the current 
zoom is interpolated from the curve and applied to all transformed two-dimensional 
pixel positions points. The distortion data can also be obtained from the lens 
10 manufacturer or can measured by someone skilled in the art. 

The foregoing detailed description of the invention has been presented for 
purposes of illustration and description. It is not intended to be exhaustive or to 
limit the invention to the precise form disclosed, and obviously many modifications 
and variations are possible in light of the above teaching. The described 
15 embodiments were chosen in order to best explain the principles of the invention 
and its practical application to thereby enable others skilled in the art to best utilize 
the invention in various embodiments and with various modifications as are suited 
to the particular use contemplated. The invention is, thus, intended to be used with 
many different types of live events including various sporting events and non- 
20 sporting events. It is intended that the scope of the invention be defined by the 
claims appended hereto. 
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CLAIMS 



1 1 . A system for using attitude sensors with a camera, said camera being 

2 part of a camera assembly, said camera assembly including a fixed portion and a 

3 movable portion, said system comprising: 

4 a first sensor coupled to said camera assembly, said first sensor measures 

5 movement of said movable portion relative to said fixed portion; and 

6 a first inclinometer coupled to said camera assembly, said first inclinometer 

7 measures attitude information of at least a portion of said camera assembly. 

1 2. A system according to claim 1, wherein: 

2 said first sensor measures rotation of said movable portion about a first axis. 

1 3. A system according to claim 2, wherein: 

2 said first inclinometer measures a component of the movement of said first 

3 axis. 

1 4. A system according to claim 1, wherein: 

2 said first sensor is an optical encoder. 

1 5. A system according to claim 1, further comprising: 

2 a second inclinometer coupled to said camera assembly, said first 

3 inclinometer and said second inclinometer are used to measure an orientation of said 

4 camera assembly. 
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1 6. A system according to claim 5, wherein: 

2 said first encoder is coupled to said movable portion; 

3 said first inclinometer is coupled to said fixed portion; and 

4 said second inclinometer is coupled to said fixed portion. 

1 7. A system according to claim 5, wherein: 

2 said first sensor measures panning of said camera; 

3 said first inclinometer measures roll of said fixed portion; and 

4 said second inclinometer measures pitch of said fixed portion. 

1 8. A system according to claim 5, wherein: 

2 said first sensor measures tilting of said camera; 

3 said first inclinometer measures roll of said fixed portion; and 

4 said second inclinometer measures pitch of said fixed portion. 

1 9. A system according to claim 5, wherein: 

2 data from said first sensor is combined with data from said first inclinometer 

3 and said second inclinometer in order to describe said camera's orientation. 

1 10. A system according to claim 5, wherein: 

2 data from said first sensor is combined with data from said first inclinometer 

3 and said second inclinometer, said combined data is used to transform a location in 

4 a first coordinate system to a position in a second coordinate system. 

1 1 1 . A system according to claim 5, further comprising: 

2 a second sensor coupled to said camera assembly, said first sensor measures 
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3 movement of said movable portion about a first axis and said second sensor 

4 measures movement of said movable portion about a second axis, said first 

5 inclinometer and said second inclinometer measure movement of said first axis and 

6 said second axis. 

1 12. A system according to claim 1, further comprising: 

2 one or more processors, said one or more processors receive data from said 

3 first inclinometer and said first sensor, said one or more processors programmed 

4 to determine attitude parameters describing an orientation of said camera based on 

5 said data from said first inclinometer and said first encoder. 

1 13. A system according to claim 12, wherein: 

2 said one or more processors use said attitude parameters to transform a 

3 location in a first coordinate system to a position in a second coordinate system. 

1 14. A system according to claim 1, wherein: 

2 said fixed portion includes a tripod and a tripod head interface; 

3 said movable portion includes at least a portion of a tripod head and said 

4 camera; 

5 said first sensor is coupled to said tripod head; 

6 said first inclinometer is coupled to said tripod head interface. 

1 15. A system according to claim 1, further comprising: 

2 sensor electronics located with said camera assembly and in communication 

3 with said first sensor and said first inclinometer, said sensor electronics reads data 

4 from said first sensor and said first inclinometer and packages said data for 
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5 transmission to a processor. 

1 16. A system according to claim 15, wherein: 

2 said sensor electronics encodes said packaged data for transmission in an 

3 audio signal to a first location, said processor being located at said first location. 

1 17. A system according to claim 1, further comprising: 

2 a second inclinometer coupled to said camera assembly, said first 

3 inclinometer is mounted in a first plane, said second inclinometer is mounted in a 

4 second plane, said first plane being orthogonal to said second plane; 

5 a second sensor coupled to said movable portion, said first sensor and said 

6 second sensor are optical encoders, said first sensor measures rotation of said 

7 movable portion about a first axis, said second sensor measures rotation of said 

8 movable portion about a second axis, said first and second inclinometers measure 

9 movement of said first axis and said second axis; 

10 a processor programmed to combine data from said first inclinometer, said 

1 1 second inclinometer, said first sensor and said second sensor in order to describe an 

12 orientation of said camera, said processor is in communication with said first 

13 inclinometer, said second inclinometer, said first sensor and said second sensor. 

1 18. A system according to claim 17, further comprising: 

2 a first gyro in communication with said processor; and 

3 a second gyro in communication with said processor, said processor 

4 combines data from said first gyro and said second gyro with data from said first 

5 inclinometer, said second inclinometer, said first sensor and said second sensor 
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1 19. A method for using attitude sensors with a camera, said camera 

2 being part of a camera assembly, said camera assembly including a fixed portion and 

3 a movable portion, said system comprising: 

4 sensing data from a first sensor, said first sensor measures movement of said 

5 movable portion relative to said fixed portion; 

6 sensing data from a first inclinometer, said first inclinometer measures 

7 absolute attitude information of at least a portion of said camera assembly; and 

8 combining said data from said first sensor with said data from said first 

9 inclinometer. 

1 20. A method according to claim 19, wherein said step of combining 

2 includes: 

3 creating one or more transformation matrices using said data from said first 

4 sensor and said data from said first inclinometer. 

1 21. A method according to claim 19, further comprising the step of: 

2 selecting a location in a scene; 

3 converting said location to a position in a video image from said camera, 

4 said step of converting is based on said step of combining; and 

5 adding a graphic to said video image from said camera at said position. 

1 22. A method according to claim 19, wherein: 

2 said first sensor measures rotation of said movable portion about a first axis; 

3 and 

4 said first inclinometer measures a component of the orientation of said first 

5 axis. 
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1 23. A method according to claim 19, further comprising the step of: 

2 adding said data from said first sensor and said first inclinometer to an audio 

3 signal for transmission to a first location, a first processor is located at said first 

4 location, said first processor performs said step of combining. 

1 24. A method according to claim 19, further comprising the step of: 

2 sensing data from a first gyro, said step of combining includes combining 

3 said data from said first gyro with said data from said first sensor and said data from 

4 said first inclinometer 

1 25 . A system for using attitude sensors with a camera, said camera being 

2 part of a camera assembly, said camera assembly including a fixed portion and a 

3 movable portion, said system comprising: 

4 a first sensor coupled to said camera assembly, said first sensor measures 

5 movement of said movable portion with respect to said fixed portion; and 

6 a first gyro coupled to said camera assembly, said first gyro measures 

7 attitude information of at least a first portion of said camera assembly. 

1 26. A system according to claim 25, wherein: 

2 said gyro is a fiber optic gyro. 

1 27. A system according to claim 25, wherein: 

2 said first sensor measures rotation of said movable portion about a first axis . 
1 28. A system according to claim 27, wherein: 
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2 said first gyro measures movement of said first axis. 

1 29. A system according to claim 25, further comprising: 

2 a second gyro coupled to said camera assembly, said second gyro capable 

3 of measuring attitude information of at least said portion of said camera assembly, 

4 data from said first gyro, said second gyro and said first sensor is combined to 

5 describe an orientation of said camera. 

1 30. A system according to claim 25, further comprising: 

2 a processor, said processor receives and combines data from said first sensor 

3 and said first gyro to describe an orientation of said camera. 

1 3 1 . A system according to claim 25, further comprising: 

2 a first inclinometer coupled to said camera assembly; and 

3 a processor, said processor receives and combines data from said first 

4 sensor, said first gyro and said first inclinometer to describe an orientation of said 

5 camera. 

1 32. A system according to claim 25, further comprising: 

2 a first inclinometer coupled to said camera assembly, said first inclinometer 

3 capable of measuring attitude information in a first plane for said camera assembly; 

4 a second inclinometer coupled to said camera assembly, said second 

5 inclinometer capable of measuring attitude information in a second plane for said 

6 camera assembly, said first plane is different from said second plane; 

7 a second gyro coupled to said camera assembly, said second gyro capable 

8 of measuring attitude information in a third plane for at least said portion of said 
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9 camera assembly, said first gyro measures attitude information in a fourth plane for 

10 at least said portion of said camera assembly, said third plane is different from said 

1 1 fourth plane; and 

12 a second sensor coupled to said camera assembly, said first sensor measures 

13 movement of said movable portion with respect to said fixed portion along a first 

14 axis, said second sensor measures movement of said movable portion with respect 

15 to said fixed portion along a second axis different, said first axis is different from 

16 said second axis. 

1 33. A system according to claim 32, further comprising: 

2 one or more processors receiving and combining data from said first gyro, 

3 said second gyro, said first inclinometer, said second inclinometer, said first sensor 

4 and said second sensor; 

5 said one or more processors use said combined data to add a graphic to a 

6 video image from said camera at a first position in said video image, said graphic 

7 corresponds to a three dimensional location within a field of view of said camera, 

8 said three dimensional location corresponds to said first position in said video 

9 image. 

1 34. A system according to claim 33, further comprising: 

2 an audio signal generator in communication with and receiving sensor data 

3 from said first gyro, said second gyro, said first inclinometer, said second 

4 inclinometer, said first sensor and said second sensor, said audio signal generator 

5 creates a compatible audio signal which includes said sensor data, said audio signal 

6 generator communicates said compatible audio signal to said camera for 

7 transmission in said camera audio signal; and 
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8 a data extractor receiving said camera audio signal and extracting said 

9 sensor data, said data extractor in communication with said one or more processors. 

1 35. A system according to claim 25, further including: 

2 a first circuit for compensating for offset of said first gyro. 

1 36. A system according to claim 25, further including: 

2 a first circuit for reducing error due to drift in said first gyro. 

1 37. A method for using attitude sensors with a camera, said camera 

2 being part of a camera assembly, said camera assembly including a fixed portion and 

3 a movable portion, said system comprising: 

4 sensing data from a first sensor, said first sensor measures movement of said 

5 movable portion relative to said fixed portion; 

6 sensing data from a first gyro, said first gyro measures attitude information 

7 of at least a portion of said camera assembly; and 

8 combining said data from said first sensor with said data from said first gyro. 

1 3 8 . A method according to claim 3 7, further including the step of: 

2 reducing errors due to drift in said first gyro. 

1 39. A method according to claim 37, further comprising the step of: 

2 selecting a location in a scene; 

3 converting said location to a position in a video image from said camera, 

4 said step of converting is based on said step of combining; and 

5 adding a graphic to said video image from said camera at said position. 
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1 40. A method according to claim 37, further comprising the step of: 

2 sensing data from a second gyro, said step of combining includes combining 

3 said data from said second gyro with said data from said first sensor and said data 

4 from said first gyro. 

1 41 . A method according to claim 37, further comprising the step of: 

2 adding said data from said first sensor with said data from said first 

3 inclinometer to an audio signal for transmission to a first location, a first processor 

4 is located at said first location, said first processor performs said step of combining. 

1 42. A method according to claim 37, further including the step of: 

2 compensating for offset of said first gyro. 

1 43. A method for using camera attitude sensors with a camera, the 

2 method comprising the steps of: 

3 sensing camera attitude information for said camera using a first set of one 

4 or more camera attitude sensors; and 

5 transmitting said camera attitude information as an audio signal to one or 

6 more processors. 

1 44. A method according to claim 43, wherein: 

2 said camera has a video signal output, an audio signal output and an audio 

3 signal input; and 

4 said step of transmitting includes communicating said camera attitude 

5 information to said audio signal input. 
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1 45. A method according to claim 43, further comprising the step of: 

2 encoding said camera attitude information onto audio or video signal prior 

3 to transmitting. 

1 46. A method according to claim 43, further comprising the step of: 

2 removing said camera attitude information from said audio signal. 

1 47. A method according to 46, further comprising the step of: 

2 using said camera attitude information to add a graphic to a video from said 

3 camera, said step of using being performed subsequent to said step of determining. 

1 48. A system for using attitude sensors with a camera, said camera 

2 having a camera video signal and a camera audio signal in communication with 

3 remotely located production equipment, the system comprising: 

4 a first camera attitude sensor; and 

5 an audio signal generator in communication with said first camera attitude 

6 sensor, said audio signal generator creates an audio signal which includes data from 

7 said first camera attitude sensor, said audio signal generator communicates said 

8 audio signal for transmission to said remotely located production equipment. 



49. A system according to claim 48, wherein: 

said audio signal generator includes a modulation circuit. 

50. A system according to claim 48, further comprising: 

a data extractor receiving said audio signal and extracting said data from 
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said first camera attitude sensor, said data extractor is part of said remotely located 
production equipment. 



1 



51. A system according to claim 50, wherein: 
said data extractor includes a demodulation circuit. 



2 



1 



52. A system according to claim 48, wherein: 

said camera includes an audio input and an audio output; 



2 



3 said audio signal generator is in communication with said audio input; and 

4 said audio output is in communication with said remotely located production 

5 equipment. 

1 53 . A system according to claim 48, wherein: 

2 said first camera attitude sensor is an inclinometer. 
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ABSTRACT 



A system is disclosed for using camera attitude sensors with a camera. A 
camera assembly includes a tripod base, a tripod head interface mounted on the 
tripod base, a tripod head mounted on the tripod head interface and a camera 
mounted on the tripod head. The tripod head enables the camera to pan and tilt. 
The system also includes a first optical encoder for detecting the amount that the 
camera has been panned and a second optical encoder for detecting the amount that 
the camera has been tilted. Two inclinometers are mounted on the tripod head 
interface to measure attitude of the tripod head. Two gyroscopes ("gyros") are 
mounted on the camera assembly. Data from the encoders, gyros and inclinometers 
are packaged and sent to graphics production equipment to be used for enhancing 
video captured by the camera. 
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